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LIST OF CHANGES TO COMMITTEE MEMBERSHIP 


MARIA VAN BOMMEL was replaced by WAYNE ARTHURS on September 2292010. 
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PREAMBLE 


The Literacy and Numeracy Secretariat (Secretariat), part of the Ministry of 
Education’s Student Achievement Division, was established in 2004 to provide 
specific support to school boards and schools.’ The Secretariat is to assist them in 
~ fulfilling their responsibilities to improve the achievement of children from junior 
kindergarten to grade six and to help close gaps in achievement for lower- 
performing student groups and schools. Achievement is measured by Education 
Quality and Accountability Office (EQAO) test results in reading, writing, and 
mathematics. ; 


The Auditor General (Auditor) noted that although the Ministry of Education. 
(Ministry) did hot achieve its EQAO target that 75% of 12-year-olds by 2008 
would achieve a level-three or B average standard, substantial progress had been 
made over the last five years. The number of children achieving level three 
increased on average from 56% in 2003/04 to 65% in 2007/08. The Auditor said 
that further increasing this percentage will be challenging and made a number of 
suggestions for the Secretariat. The Auditor said, for example, Secretariat 
program funding was not always allocated to school boards and schools with the 
greatest need. He also said that neither the Secretariat nor the school boards 
visited adequately documented, monitored, or reported on school board 
improvement plans to improve student achievement. These are some of the 
findings of the Auditor’s value-for-money audit of the Literacy and Numeracy 
Secretariat (section 3.07 of the Auditor’ General’s 2009 Annual Report). 


In May 2010 the Standing Committee on Public Accounts held public hearings on 
the Auditor’s report. Senior officials from the Ministry and the Secretariat (which, 
as noted earlier, is part of the Ministry) participated in the hearings. (For a 
transcript of the Committee proceedings please see Committee Hansard, May 19, 
2010.) The Committee endorses the Auditor’s findings and recommendations. 
This Committee report presents the Committee’s findings, views, and 
recommendations. The Committee requests that the Ministry and the Secretariat 
provide the Committee Clerk with written responses to the recommendations 
within 120 calendar days of the tabling of the report with the Speaker of the 
Legislative Assembly, unless otherwise specified in a recommendation. 


Acknowledgments 


The Standing Committee on Public Accounts extends its appreciation to officials 
from the Ministry of Education, including the Literacy and Numeracy Secretariat, 
for their attendance at the hearings. The Committee also acknowledges the 

~ assistance provided during the hearings and report writing deliberations by the 
Office of the Auditor General, the Clerk of the Committee, and staff in the 
Legislative Research Service. 


' The Secretariat, when created, was associated with but separate from the Ministry of Education. 





OVERVIEW 


The Secretariat works with over 4,000 elementary schools across 72 school 
boards. It serves English- and French- language schools in both public and 
Catholic school boards and collaborates on measures such as setting targets. The 
Secretariat employs over 80 Student Achievement Officers (SAOs) who are 
experienced educators that work with schools and school boards on education 
strategies. Since its establishment the Secretariat has spent $340 million, with 
almost $288 million transferred to school boards to help them meet student- 
achievement targets. The Secretariat has 102 employees and a budget of $81 
million. 


The Ministry said that Ontario is recognized as a global leader in education. Other 
countries look to it for advice and guidance. In international testing such as the 
Progress in International Reading Literacy Study (PIRLS), Ontario has improved 
at a faster rate than many other jurisdictions. In 2008/09, 67% of grade three and 
six students were achieving at or above the level-three standard in EQAO testing. 


Objectives and Scope of the Audit 


The objective of the audit was to assess whether the Secretariat had adequate 
procedures in place to: 


» measure and report on the effectiveness of its activities in fulfilling its 
mandate to ensure that students in Ontario achieve a aes level in reading, 
writing, and mathematics by age 12; and 


« ensure that its transfer payments to school boards are properly managed and 
directed to the areas of greatest need of support for students’ achievement 
levels to improve. | 


As part of the audit, the Auditor interviewed supervisory officers and principals at 
one French-language school board and five English-language school boards. 


ISSUES RAISED IN THE AUDIT AND BEFORE THE COMMITTEE 


Significant issues were raised in the audit and before the Committee. The 
Committee attaches particular importance to those issues discussed below. 


Measuring and Reporting on Program Effectiveness 


The Auditor noted that the Ministry’s priorities are to attain high levels of student 
achievement, close gaps in student achievement, and maintain high levels of 
public confidence. In the five years preceding his 2009 audit, average EQAO 
results for both English- and French-language schools increased by almost 10% 
(from 56% to 65%). French-language schools achieved the Ministry’s goal for 
achievement levels for 12 year olds in 2007/08; English-language schools did not. 


The Auditor noted that other jurisdictions track the test scores of individual 
students over time, for example, by comparing a particular group of students’ 
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grade three results with their grade six results three years later (cohort tracking). 
The Auditor believes that if the Ministry undertook cohort tracking in conjunction 
with test-score analysis it would have a better measure of the value of its 
investment in programs. The Auditor also noted that Ontario does not report how 
wide the gap is between the highest and lowest performers, that other jurisdictions 
do measure this and that those with narrower gaps tend to have higher average 
scores. The Auditor recommended that the Ministry report changes in the gap 
between top-performing and lower-performing student groups and schools, as 
well as how specific student cohorts perform over time while participating in the 
programs and initiatives intended to improve their performance. 


CLLRNet Report 


The Ministry referred to the Canadian Language and Literacy Research Network 
(CLLRNet) 2009 report titled The Impact of the Literacy and Numeracy 
Secretariat: Changes in Ontario’s Education System. The report said that the 
Secretariat has had a major, and primarily positive, impact on Ontario’s education 
system; there has been a significant shift in the culture of Ontario schools that is 
focused on enabling the success of all students; and that there has been sustained 
improvement in student achievement. The report also states that increased 
attention to evidence, research, evaluation and data can be expected to provide 
general, long-term benefits, across all areas of the education system in Ontario. 


Improvement in EQAO Test Score Results 


The Ministry attributed improvement in EQAO test score results to its multi- 
faceted approach to student success. It said that SAOs are respected 
superintendents, principals and classroom teachers from across Ontario who work 
directly with schools and school boards. They build knowledge and develop 
capacity to implement evidence-informed strategies to improve students’ reading, 
writing and math skills. 


The Ministry said that the Secretariat’s initiatives work in an integrated way to 
improve student learning and achievement and narrow the gaps. It is difficult to 
quantify the exact degree of impact of individual strategies. Evidence that the 
Secretariat’s approach works exists in the analysis of overall outcomes. 


According to the Secretariat, some interesting results underlie the EQAO 
improvement scores for students achieving level three and four results, For 
example, when assessing the grade six writing results the Secretariat does not just 
assess who achieves level three or four but also: 


- How many students have very high levels of special needs and are therefore 
exempt from the tests? (Over the last five-to-six years exemptions have 
decreased from 5% to 3%.) 


- Who was not able to complete enough of the test to reach level 1? (Over the 
last five-to-six years the number has decreased from 5% to less than 1%.) 
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» What proportion of students is at level one or two and how has this number 
changed over the years? (Over the last six years the number of students 
producing work at level one has decreased from 5% to less than 1%.) 


English- and French-language School Student Outcomes 


The Committee asked why French language schools had better EQAO test results 
on average than English language schools. The Secretariat said that while it is 
almost impossible to say which factor has a particular impact, French school 
boards, for the most part, have had full-day learning in junior and senior 
kindergarten for a number of years. The Secretariat noted that this can have a 
significant impact, particularly on grade three results, on building that foundation 
for learning that carries through into later years. The French school systems have 
historically worked more closely together and shared best practices. There is also 
a focus on improving outcomes for children in the French system because parents 
must consciously choose to enrol their children in it. : 


The Secretariat said that the French first-language schools and the French boards 
across the province are designed for students who have French-language rights 
under the Charter. Not all those children, however, speak French in the home. The 
Secretariat said that in the early year programs in the French school system there 
is a focus on oral language development, a critical component to learning. The 
Secretariat has been working with a number of northern school boards on an oral 
language program (international researchers are in these schools). According to 
the Secretariat, a significant number of the 12 French boards are doing well, but 
for many of the small French school boards in the North, achieving good results is 
as much of a challenge as it is for English school boards in the North. 


Gap Between Lowest-performing and Highest-performing 


Ontario reports the proportion of students who achieve at levels one, two, three, 
and four on the EQAO tests. The Secretariat said that in a sense, the gap between 
the highest and lowest students is the difference between a student performing at 
level four and a student who has not yet reached level one. According to the 
OECD, Ontario has about half the gap of many other countries in the norm. 


The Secretariat measures gaps for specific groups and tracks results. From 2002 
to 2008/09 it reduced the gap of English-language learners’ performance by half, 
which the Secretariat said is good for Ontario and is recognized positively by 
international groups, given the number of students coming to Ontario from all 
over the world. The Secretariat said that overall performance has improved but 
that the rate of improved performance of English-language learners has been 
twice that of the overall rate. It has also significantly reduced the gap for students 
with special needs, attributing this to the precision of teaching and the 
Secretariat’s ability to take one teacher’s good practice and to turn it into a 
broader good practice (across the system) quicker than ever before. The 
Secretariat added, however, that it is not able to designate the gap in performance 
on an individual basis. The Ministry said that while its cohort tracking may not be 


as exact as that in some other jurisdictions, it shows by percentage and by board, 
the difference between how many students are achieving at each level. , 


Future areas of focus will include the socio-economic gap. Schools in the Ontario 
Focused Intervention Partnership (OFIP) one and two programs were likely to be 
those schools that had, for example, socio-economic challenges or students with 
special needs challenges. The Secretariat has a Schools on the Move program 
which highlights schools that have raised student achievement and sustained 
progress in raising those achievement levels over several years. In June 2010, the 
Secretariat celebrated schools in the program facing challenging socio-economic 
circumstances to demonstrate that demographics are not destiny (see Monitoring 
and Funding of Program Initiatives section below for more information on OFIP, 
Schools on the Move and other programs). 


Committee Recommendation 


The Standing Committee on Public Accounts recommends that: 


1. The Literacy and Numeracy Secretariat shall report to the 
Standing Committee on Public Accounts on measures under 
consideration by the Secretariat to better utilize cohort tracking 
to assess the progress of the same group of students over time. 


School Board Improvement Plans 


These plans are intended to help teachers, principals and senior school board staff 
plan and implement strategies to improve student achievement in both the short 
and long terms. The Ministry has adopted SMART goals (specific and strategic, 
measurable, achievable, result-based, and time-bound) for improvement plans and 
recommended that boards develop plans based on these goals, Although not 
required, all 72 boards have been submitting their plans to the Ministry. The 
Auditor noted, among other factors, that there was little documentation that the 
Secretariat reviewed the plans and a lack of documentation on SAO feedback to 
school boards on the plans. 


The Auditor also noted that the Ministry has only informal accountability 
mechanisms in place and little assurance (or documentation) that school boards 
meet the goals or timelines in their improvement plans or undertake necessary 
corrective action. The Auditor said that more detailed board improvement plans 
that assess whether goals have been achieved, with recommended actions when 
goals are not met, would assist the Secretariat in its review of the results of school 
board program strategies. The Auditor’s recommendations included that the 
Ministry implement a formal improvement-plan review process, require school 
boards to post improvement plans online, and properly document the results of 
Ministry monitoring efforts. 


Nature of the Improvement Plans 


The Ministry said that the board improvement plan is a living document that is 
designed to plan specific student achievement goals on an annual basis as 





developed through the board’s analysis of needs assessment data. It also improves 
achievement for all students in the board and provides a tracking and monitoring 
plan for improving student achievement. Finally, the plan provides an evaluation 
of the board’s progress in meeting its goals. 


The Ministry said that one of the key reasons for Ontario’s success in improving 
student learning and achievement is the enormous support in place for educators, 
as well as the right amount of pressure and the working partnership that the 
Secretariat has established with boards. This partnership has enabled effective © 
implementation of initiatives and monitoring of results that have contributed to 
overall student improvement in outcomes. 


Completion and Review of Improvement Plans 


The Secretariat works in partnership with the school boards and schools to 
improve student outcomes. Back in 2006, the Secretariat told school boards they 
needed specific plans to improve student achievement. The school boards then 
produced plans. Douglas Reeves, a consultant from the Centre for Leadership at 
Harvard University, was engaged in 2008 to review the 72 board improvement 
plans and processes in Ontario. That review yielded recommendations for each 
board and formed the basis for board improvement plan processes. 


The Secretariat CEO said that when she was a school board director she thought 
she had good school and board improvement plans in place. Then she received a 
27-page detailed report, following the Douglas Reeves review, specifying the - 
need for further action and greater detail on what the board would change and 
how it would monitor progress. Her board “went back to the drawing board” in 
response to the Reeves report. 


In June 2009, during the most recent cycle of improvement plans, the Secretariat 
asked school boards to submit their school improvement plans. The Secretariat 
reviewed the plans over the summer, developed a template for a good plan, and 
provided each school board with feedback on its plan and the overall board 
improvement plan in place during regional meetings. 


School boards were asked to update and resubmit their plans to the Secretariat in 
October 2009. Those resubmitted plans became the basis for the Secretariat’s 
conversations with school boards throughout this past school year, and in 
particular became the foundation for discussions with them about accountability 
and monitoring changes. In January 2010, the Ministry held mid-year meetings 
with all boards to determine the progress made toward improved student 
achievement. Boards said that they focused on instruction, capacity-building, 
building school networks and the use of data to guide and inform their practices, 
set priorities and allocate resources, 


The Secretariat noted that in accordance with the Auditor’s recommendation, it 
provided better documentation on both the October review and January meetings 
with the school boards, noting a lack of adequate documentation in the past. It 
will continue to strengthen such documentation. 


The Secretariat said that the school boards now cite specific and measurable goals 
and targets. They also report how they monitor activities, results, and gains. The 
Ministry added that the overall tenor of Bill 177, the Student Achievement and 
School Board Governance Act, 2009 is that boards must be fiscally responsible 
but that they also have a core responsibility to concentrate on student achievement 
and student wellness. Boards must also develop multi-year strategic plans to 
achieve specified student achievement and well-being targets as provided for 
under Bill 177. Boards must hold their administrators responsible for student 
achievement. 2 


Monitoring and Reporting the Achievement of Plans 


Legislative and regulatory controls in place include a requirement for a director’s 
annual report that is to be posted on board websites by January 31 of each year. 
The Ministry said that all 72 boards have posted their reports which can be 
accessed through the Ministry’s website. Boards also produce annual reports with 
required information such as EQAO results and specific actions pursued to 
improve outcomes for students with low levels of performance. 


Boards must be transparent within their communities about student achievement 
goals, to report on progress in reaching the goals, and to be held accountable not 
only by the Ministry but also by parents and the school community. The Ministry 
said this is being achieved through online postings of the board reports and 7 
directors’ reports and by having province-wide testing with published results. The 
Ministry is also ensuring that financial accountability mechanisms are in place 
(see Program Funding section below for details). 


Committee Recommendation 


The Standing Committee on Public Accounts recommends that: 


2. The Ministry of Education shall report back to the Standing 
Committee on Public Accounts on criteria used by the Literacy 
and Numeracy Secretariat and the Ministry to assess school 
board improvement plans. The Ministry should include 
information on the template that it has developed for a good 
school board improvement plan. It should also explain how it 
transmits best practices for the plans to all boards. 


Monitoring and Funding of Program Initiatives 


The Secretariat offers a number of programs including, as noted earlier: 


- Ontario Focused Intervention Partnership (OFIP): Schools identified as 
low-performing or static are given targeted support, such as allowing teachers 
time away from the classroom to participate in professional development 
activities that will help them increase their effectiveness. 
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« Schools on the Move: Schools that have made substantial progress in raising 
student achievement and sustained this progress over several years are 
highlighted to share successful practices with other schools. 


Others programs listed by the Ministry include OFIP Tutoring, Schools in the 
Middle, the Character Development Initiative, the School Effectiveness — 
Framework, and various professional development programs for educators. The 
Ministry added that schools and school boards in varying circumstances benefit 
from differing levels and types of supports. It said that an educational advisor, 
Michael Fullan, is on a part-time contract with the Ministry to provide advice to 
the Secretariat on program development. 


The Ministry cited OFIP as a key initiative which the Secretariat analyzes each 
year for effectiveness in improving student achievement. Since its inception in 
2006/07, $25 million has been invested each year in OFIP. An additional $8 
million has been provided for OFIP tutoring initiatives (before and after school) 
for students in need. 


The research department produced a report on the Secretariat’s work in schools 
that are categorized as OFIP one (34% or fewer students reaching the provincial 
benchmark) and OFIP two (34%-50% of students reaching the provincial 
benchmark). When the EQAO test results for these schools were compared to the 
overall provincial results, the OFIP one and two schools demonstrated far in 
excess of the Ontario school average improved achievement. 


The Ministry said the Schools on the Move program increases confidence in 
publicly funded education by providing examples of schools that are improving 
student learning and achievement and reducing gaps in targeted groups. The 
program, which began in 2006 as a network of 23 schools, now includes over 140 
schools across the province. The Secretariat said that it knows that improving 
outcomes for children is urgent and that successful programs must be scaled up — 
quickly to schools across the province. There is no time for a two- or three-year 
analysis. The Secretariat must work with information and data on an annual basis. 
The data is both quantitative and qualitative. 


Collaborative Inquiry into Mathematics 


In 2008, the Secretariat began to focus on mathematics learning in schools. It 
developed a program based on evidence and research, implemented it in 12 school 
boards, worked with the boards for a year, and had an independent university- 
based researcher follow up with surveys to obtain qualitative feedback back from 
teachers, boards and schools involved in the program. The Secretariat concluded 
from EQAO results the following August that in 11 of the 12 boards mathematics 
improvement was greater than the provincial average and deemed the program a 
promising practice. The program was scaled up to include 24 boards and the 
Secretariat may further expand it, depending on results. 


Change Implementation 


The Secretariat noted that change implementation is not easy. It added that 
Ontario is the only jurisdiction that can show a track record of sustained year- 
over-year improvement on a system-wide basis. There are not many researched 
examples of this kind of systematic change. The Secretariat noted a fundamental 
culture shift in teaching and learning and added that the result has been an open, 
collaborative culture which is key to teachers and principals owning the changes 
taking place, which in turn makes the changes sustainable. 


The Secretariat said that high-quality teaching focused on individual children and 
implementing changes in teacher practices in order to meet the needs of those 
children are crucial. It noted that there is uneven implementation at best; 
implementing change in classrooms has not been an area of great success for the 
Ministry or government until recently. 


The Secretariat stressed that at the school level, programs such as OFIP are really 
means of increasing the capacity of teachers and principals to implement 
measures for children that evidence and research indicate make a difference, and 
facilitate accurate and fast sharing of information across the province. Once 
teachers understand there is something that they can change in their practice to 
improve outcomes for their students, they will implement that change. 


Resources for Teachers 


The Secretariat said that it provides resources directly to teachers such as DVDs 
and monographs on practices that work well. In addition, in 2010 it focused on 
partnering with teachers to help children achieve level two EQAO results. For 
example, the Secretariat has worked with the Durham board on a program where a 
teacher works in classrooms in several schools that have large numbers of 
students producing work at level two, focusing, for example, on the kind of 
feedback teachers need to provide those students. Lessons learned are shared with 
other teachers. 


The Secretariat said that it provides funding for teacher meetings, where teachers 
bring and examine student work, in order for teachers to be able to compare their 
assessment and expectations of students. One goal is to determine how teachers 
transmit those expectations to students. The Secretariat said that teaching is a very 
personal act with differences in styles but that the Secretariat is helping to 
promote consistency in assessments, expectations and tasks given to children. 


Reducing Class Sizes 


The Secretariat said that smaller pupil-teacher ratios can be beneficial but also 
noted that in more isolated schools, such as some in the north that have lower 
ratios, teachers lack other support mechanisms. For example, a teacher in this 
setting may be the only grade three or four teacher in the school and therefore 
unable to share experiences with a colleague doing the same job. 
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The Secretariat also noted research which indicates that if class size reduction 
improves outcomes for student learning, it is more likely to be in the primary 
division than elsewhere. It added that many large-scale attempts to reduce class 
sizes, such as in California, resulted in lower quality learning for children because 
of a lack of highly qualified, professional teachers (needed to teach in a larger 
number of smaller classes). The Secretariat noted differences in its approach in 
comparison with other jurisdictions. Ontario implemented class size reductions 
over a three-year period. This provided time to be selective in hiring the extra 
teachers required and occurred when the Secretariat was already investing 
significantly in building teacher capacity. 


The Ministry said that it has clear evidence that in recent years the quality of 
teaching (already good) has improved across Ontario. It said there is now a 
process for teacher performance appraisal with accountability provisions. Citing 
data from five years ago, the Ministry said that there were six applicants for each 
spot available at teachers’ colleges so the colleges were able to choose from “the 
best and the brightest.” 


Monitoring Student Achievement Initiatives 


The Auditor said that in order for the Secretariat to have timely interventions and 
effective programs it must know which program initiatives work best and which 
should be modified or eliminated. He noted that the school boards that he visited 
‘had not carried out sufficient assessments of Secretariat initiatives and how these 
contributed toward improving student achievement (he did note that OFIP is 
assessed annually). The Ministry, as noted earlier, said that it is the sum effect of 
all programs that contributes to student achievement and noted difficulties in 
evaluating each program individually. The Auditor acknowledged the Ministry’s 
position but recommended that the Secretariat should formally evaluate how well 
all its program initiatives contribute to improving student achievement, and 
modify or eliminate the less effective initiatives. 


The Ministry said that it will review its programs over a three-year period, adding 
that the review process needs to be continual. As circumstances change, programs 
will need to change. The Ministry said that the Secretariat must be “fleet of foot,” 
but that program documentation must also keep pace. The Ministry completed its 
RFP process in spring 2010 in order to establish vendors of record able to 
undertake formal program evaluations. During the first year, the Ministry will 
review a mix of large programs, such as OFIP, and smaller ones, such as small 
northern boards. It will review a similar mix over the subsequent two years of the 
three-year review process. 


Committee Recommendation 


The Standing Committee on Public Accounts recommends that: 
3. The Ministry of Education shall report back to the Standing 


Committee on Public Accounts on the Ministry’s review of the 
effectiveness of the various Literacy and Numeracy Secretariat 
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programs. The Ministry should specify the review criteria, the 
results of reviews already conducted, and any changes to 
programs that have resulted from the reviews. 


Program Funding 


The Auditor said that some program funding was based on need; in some cases 
schools received a set amount; and in other cases the Secretariat could not fully 
explain the funding allocation model. According to the Auditor, funding that is 
based on average daily enrolment as opposed to relative need fails to focus scarce 
resources on the highest priority areas identified by assessments such as EQAO 
testing and fails to target funding to low-performing schools or boards. He noted 
that not all school boards submitted reports on the use of program funds to the 
Secretariat and there was little follow-up to obtain these reports. He recommended 
that the Secretariat should ensure its program funds are allocated to the areas of 
greatest need and that the funds are being spent for the intended purpose. 


The Secretariat said that from 2006/07 to 2009/10 half of the $25-million in 
funding was allocated based on enrolment and the other half on need. The Auditor 
said there was no documentation that demonstrated this at the time of his audit 
and that it appeared to the Auditor at that time as though enrolment played a 
significant role in determining the allocation of funds. The Secretariat said that it 
now has a better breakdown and clear data about socio-economic challenges to 
schools. This data is matched against overall results for schools. 


‘Needs Based Funding: Schools in the Middle 


The Ministry said that in 2009/10 OFIP funding was adjusted to reflect the 
significant reduction in the number of low-performing schools, defined as those 
where less than 34% of students achieved level three and four on EQAO testing. 
The Ministry reduced the overall number of schools in the low-performing 
category from 19% to 6%. The reduction in OFIP funding enabled the Secretariat 
to provide funding support to “middle initiative” schools, where 50% - 74% of the 
students achieve the provincial standard on four out of the six EQAO assessments 
in grades three and six. It said this demonstrates that the Secretariat does measure 
outcomes to assess program effectiveness and make revisions as needed. 


Boards receive a base amount of funding depending on the number of schools that 
fit the “in the middle” criteria. In 2009/10, approximately $9 million was provided 
to boards for support to more than 1,400 schools in this category. 


The Secretariat said that $59 million for capacity building is support for improved 
teaching and learning. The Ministry said that it agreed with the Auditor that it is 
necessary to specify the funding formula for each of the programs, as well as the 
rationale behind the formula. The Ministry hopes to have documentation in place 
that will satisfy the Auditor when he next examines the Secretariat. 
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Needs Based Funding: Small and Northern Board Numeracy Initiative 


In 2009/10, about $2 million to $3 million was provided to 17 school boards to 
reach over 400 small and northern schools, in order to build capacity in numeracy, 
teaching and learning. Math facilitators are supporting teachers in deepening their 
understanding of effective instructional practices for mathematics. 


The Secretariat explained the process for targeting these boards. It first examined 
board level data to assess areas of greater need. It then focused on individual 
school data as well as the numbers of schools. As noted earlier, a certain amount 
of board funding is on a per capita basis. Some programs, however, are more | 
focused on schools that demonstrate they are dealing with challenges. 


According to the Secretariat, public data on lower-performing schools and actual 
results may be different. For example, in many school areas in the north the 
schools are so small that EQAO suppresses public data because it may identify 
individual students. It is possible that public data may be skewed towards low 
performance in urban areas because of this type of data suppression. The 
Secretariat has access to the private data, which helps inform assessments. ~ 


The Secretariat also said some boards need additional funding because of 
geography and the distances that people have to travel. For example, schools in 
small and northern boards are often situated far apart, which makes sharing 
information and practices more difficult. For these 17 boards the Secretariat 
directly funded an additional staff member to spend consecutive weeks in 
different schools working with individual teachers. The Secretariat said evidence 
now indicates that this classroom-embedded professional learning is having a 
greater impact on changing teacher practice than some other methods. The 
Secretariat said a large differential in the amount of per student support in two 
different boards in one example cited by the Auditor arose because the school 
with lower support was a large urban board in the south while the school with 
higher support was a northern board with a large geographic footprint. 


Funding to Boards 


The Ministry noted that school boards must report on funding received as well as 
the effectiveness of strategies and lessons learned so that appropriate adjustments 
can be made. The Ministry said that its documentation on how it spent money in 
the past was not as good as it could be. It drew attention to a number of enhanced 
accountability measures now in place. The Ministry has a formal transfer payment 
contract with each school board for funding initiatives that clearly outlines the 
Ministry’s expectations. School boards provide the Ministry with financial and 
activity reports on an annual basis. The Ministry will follow up and through 
evaluation, determine whether money spent is being used effectively and will also 
consider whether money should be redirected for better use. 


The Secretariat is also reviewing its financial processes and developing a logic 
model to address the need for improved documentation and to further define 
program objectives, outcomes, measures and reporting requirements. The logic 


i) 


model for the programs will link strategic planning to the funding process and the 
outcomes. The Ministry will undertake a review of lead board use, which it 
expects to complete by the end of fiscal year 2010/11. It has taken interim steps to 
bring many initiatives administered by the lead boards back into the Ministry. As 
noted earlier, it now has a documentation process to track how money is being 
used and will receive reports on its use. 


Committee Recommendation. 


The Standing Committee on Public Accounts recommends that: 


4. The Ministry of Education shall provide the Standing Committee 
on Public Accounts with the Ministry’s most recent data showing 
how much funding for Literacy and Numeracy Secretariat 
programs was allocated based on need and how much was 
allocated based on enrolment. The Ministry should include 
information on its criteria for determining how it assesses 

whether program money has been spent effectively and whether 
the Ministry has identified any program areas where money 
currently being allocated for certain programs should be 
redirected elsewhere. The Ministry should also report on the 
percentage of total funding for the specific programs actually 
spent on the intended services. 


Use of Financial Agents: “Banker Boards” and Lead Boards 


The Auditor noted that in some cases the Secretariat uses financial agents to act as 
distributors of ministry funds to third parties or other school boards. These agents 
include the Council of Directors of Education (CODE) and a number of school 
boards, and are collectively referred to as “banker boards.” The Auditor’s review 
of these financial arrangements generally indicated that proper accountability 
measures to effectively monitor the banker boards and ensure that government 
funds were being spent appropriately were not in place. He said some banker 
board administrative fees were far in excess of the 13% average. The Auditor 
recommended that the Secretariat ensure that the CODE expenditures are 
appropriately approved and supported and reconsider pre-flowing funds to 
“banker” school boards. 


The Secretariat will consider on a case-by-case basis when it may be more 
appropriate to use a lead board. The Ministry said, for an example, this would 
likely be appropriate for a regional meeting in Kenora because a lead board would 
have knowledge of the best and most economical facilities available. The Ministry 
added that since receiving the Auditor’s report the Secretariat has centralized and 
reviewed its financial documentation that supports the amounts paid to CODE and 
to lead boards. 


The Ministry said that part of its review of lead board use will focus on the 
administration fee paid to banker boards or lead boards. The fee covers overhead 
such as phone calls and working with other boards. The Ministry said it will 
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consider the best way to pay administration costs that is appropriate and relevant 
to the effort made and that it is not wedded to a particular percentage. The 
Ministry will also examine the administration fee charged by CODE. 


Committee Recommendations 


The Standing Committee on Public Accounts recommends that: 


5. The Ministry of Education shall provide the Standing Committee 
on Public Accounts with an interim report on the Ministry’s 
review of lead or banker board use. The Ministry should specify: 


« whether it is on track to complete the review by the end of 
fiscal year 2010/11; 


« what initiatives formerly administered by lead or banker 
boards have been returned to the Ministry; 


- whether the Ministry will continue to use lead or banker 
boards and if so, what the criteria will be for selecting the 
boards and monitoring their expenditures; and 


« what criteria the Ministry will use to determine appropriate 
levels of payment for services provided by these boards and 
how expenses submitted by the boards will be reviewed. 


Consistency of Student Assessments 


The Auditor said that in general, the only assessments made throughout the school 
year that parents see are report card marks and EQAO test scores for grades three 
and six. He noted in a 2003 audit that one possible method of measuring 
consistency in student assessments for some subjects is to compare report card 
marks to EQAO scores but said that the Ministry does not have a procedure in 
place for this. After each of his school board visits, the Auditor undertook his own 
comparison of report card marks with EQAO scores and found that for 4% of 
students, EQAO scores bear no resemblance to report card marks. The Auditor 
recommended that the Ministry should monitor the results from different types of 
assessment, especially those from report cards and EQAO tests, to identify any 
major discrepancies for follow-up. 


The Ministry said that it commissioned a study comparing student report card 
marks with grade three and six EQAO achievement results. It said that the study 
determined that there is comparability between the two different measures of 
student achievement. The Ministry said, however, that further analysis of the data 
is needed to confirm the extent of the relationship and to identify factors that 
influence that relationship. 


Referring to the Auditor’s observation that. EQAO scores bore no resemblance to 
report card marks for 4% of students, the Ministry said that standardized tests held 
on one day are not always reflective of student performance throughout the year. 
For example, a student may come to school sick on the day of the test and receive 
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a bad mark. The Secretariat added that the discrepancy is more likely to occur for 
students with particular barriers to learning. The EQAO test takes place with 
some accommodation of special needs but often does not include the kind of 
‘broad accommodations that are in place for the student throughout the year, The 
Secretariat said that: both the EQAO assessments and the report card marks are 
important assessments of where students are, and give boards, schools and 
teachers important information. — 


Committee Recommendation 


The Standing Committee on Public Accounts recommends that: 


6. The Ministry of Education shall provide the Standing Committee 
on Public Accounts with an update on the Ministry’s assessment 
of data from the study that it commissioned to compare the 
consistency of student report card marks with their grade 3 and 6 
EQAO achievement results and inform the Committee as to 
whether it will consider ongoing correlation tracking. 


Ontario Statistical Neighbours Information System (OSN) 


The Auditor described the Ontario Statistical Neighbours Information System as 
an information system for analyzing school performance, demographics, and 
school program information. The database, among other uses, is intended to help 
the Secretariat with strategic planning and to identify similar schools to facilitate 
the sharing of best practices. School principals’ use of the tool was infrequent 
because the principals did not have direct access to the system — instead, they had 
to get the information from their SAOs. The Auditor recommended that the 
Ministry should consider granting all school boards and schools direct access to 
the system. This would be more cost-effective than school boards having to 
develop and maintain their own systems, as some were doing. 


The Ministry said that it has developed an online version of the OSN database that 
provides school board superintendents with access to an information management 
system. It agrees with the Auditor that access to this system will be useful for 
developing strategies for improving student achievement at the board and school 
levels. The information will enable school boards and schools to identify whether 
their results are improving, declining or remaining static and to compare 
themselves with similar schools or boards based on demographics and other 
program information. Superintendents will be able to share this information with 
their school principals. The Ministry has been training superintendents on how to 
use the online OSN system. 


The Ministry has used the system to identify schools needing support, including a 
number of coasting schools that perform above average, but are not making 
continued gains. It has also enabled the Ministry to identify a set of “schools on 
the move.” Staff members at those schools are expected to share their knowledge 
and practices through networking with other schools across Ontario. 
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CONSOLIDATED LIST OF RECOMMENDATIONS 


The Standing Committee on Public Accounts requests that the Ministry provide 
the Committee Clerk with a written response to each of the Committee’s 
recommendations within 120 calendar days of the tabling of the report with the 
Speaker of the Legislative Assembly, unless otherwise specified in a 
recommendation. 


The Standing Committee on Public Accounts recommends that: 


1. The Literacy and Numeracy Secretariat shall report to the Standing Committee 
on Public Accounts on measures under consideration by the Secretariat to better 
utilize cohort tracking to assess the progress of the same group of students over 
time. 


2. The Ministry of Education shall report back to the Standing Committee on 
Public Accounts on criteria used by the Literacy and Numeracy Secretariat and 
the Ministry to assess school board improvement plans, The Ministry should 
include information on the template that it has developed for a good school board 
improvement plan. It should also explain how it transmits best practices for the 
plans to all boards. ) 


3. The Ministry of Education shall report back to the Standing Committee on 
Public Accounts on the Ministry’s review of the effectiveness of the various 
Literacy and Numeracy Secretariat programs. The Ministry should specify the 
review criteria, the results of reviews already conducted, and any changes to 
programs that have resulted from the reviews. 


4. The Ministry of Education shall provide the Standing Committee on Public 
Accounts with the Ministry’s most recent data showing how much funding for 
Literacy and Numeracy Secretariat programs was allocated based on need and 
how much was allocated based on enrolment. The Ministry should include 
information on its criteria for determining how it assesses whether program 
money has been spent effectively and whether the Ministry has identified any 
program areas where money currently being allocated for certain programs should 
be redirected elsewhere. The Ministry should also report on the percentage of 
total funding for the specific programs actually spent on the intended services. 


5. The Ministry of Education shall provide the Standing Committee on Public 
Accounts with an interim report on the Ministry’s review of lead or banker board 
use. The Ministry should specify: 


« whether it is on track to complete the review by the end of fiscal year 
2010/11; 


* what initiatives formerly administered by lead or banker boards have been 
returned to the Ministry; 
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« whether the Ministry will continue to use lead or banker boards and if so, 
what the criteria will be for selecting the boards and monitoring their 
expenditures; and 


» what criteria the Ministry will use to determine appropriate levels of payment 
for services provided by these boards and how expenses submitted by the 
boards will be reviewed. 


6. The Ministry of Education shall provide the Standing Committee on Public 
Accounts with an update on the Ministry’s assessment of data from the study that 
it commissioned to compare the consistency of student report card marks with 
their grade 3 and 6 EQAO achievement results and inform the Committee as to 
whether it will consider ongoing correlation tracking. 
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